AquaX Deployment Plan

Owner: DevOps
Reviewers: Backend, Frontend, Mobile, QA, Security, Operations
Status: Draft - Current-Code Baseline
Version: 0.1
Last Updated: 2026-09-17
Review Cycle: Per release, infrastructure change, or deployment workflow change

1. Muc Dich Va Pham Vi

Tai lieu nay mo ta ke hoach trien khai AquaX theo trang thai code hien tai. Noi dung duoc tong hop tu docker-compose.yml, docker-compose.iot.yml, .github/workflows/, k8s/, cac Dockerfile, cac file .env.example, script mobile va cac tai lieu DevOps/Operations dang co.

Pham vi bao gom:

• Trien khai local development.
• Trien khai local IoT stack.
• Trien khai backend API tren VM/Docker Compose theo ha tang production hien tai.
• Trien khai backend API len ST United Docker registry/server theo workflow hien co.
• K8s/GKE deployment path duoc giu nhu phuong an optional/reference vi repo van co manifests va workflow.
• Quy trinh build/release mobile Android theo script hien co.
• Cau hinh moi truong, secrets, migration, verification, rollback, monitoring va cac gap can bo sung.

Pham vi chua duoc code xac nhan day du:

• Production hosting cho web admin portal.
• Production topology cho MQTT broker, Telegraf va IoT worker.
• iOS App Store release process.
• Backup/RPO/RTO/DR production chinh thuc.
• Deployment runtime rieng cho AI engine.

2. Tom Tat Trang Thai Deployment Hien Tai

3. Deployment Topology

3.1 Local Development

Local integrated stack trong docker-compose.yml gom:

3.2 Local IoT Stack

docker-compose.iot.yml phuc vu luong IoT rieng:

• PostgreSQL/TimescaleDB tren host port 5433.
• Mosquitto tren port 1883.
• iot-migrate chay Prisma migration bang backend image.
• Telegraf ghi telemetry vao database IoT.
• IoT worker doc database local hoac forward telemetry/heartbeat/ack len cloud backend khi cau hinh IOT_WORKER_CLOUD_*.

3.3 VM/Docker Compose Backend - Production Path Hien Tai

Production infrastructure hien tai dang chay tren VM, khong phai Kubernetes. Do do deployment plan chinh cho production can uu tien:

• Build backend image tu backend/Dockerfile.prod.
• Push image len registry duoc VM truy cap, hien co registry ST United docker-gateway.stunited.vn.
• SSH vao VM hoac dung CI/CD SSH step.
• Tai thu muc deploy tren VM phai co docker-compose.yml va .env.
• Chay docker compose pull va docker compose up -d cho service API production.
• Verify container, logs va health endpoint sau deploy.

Repository hien co workflow VM cho dev:

• Workflow: .github/workflows/cd-api-stunited-registry.yml.
• Branch: dev.
• Default deploy dir: /home/stunited/st-projects/aquax/dev.
• Service: aquax-api-dev.
• Tags: dev-<date>-<sha> va dev-latest.

Production VM gap can bo sung:

• Workflow hoac runbook rieng cho production VM.
• Ten service production trong Docker Compose tren VM.
• Production deploy directory.
• Production image tag convention, vi du prod-<date>-<sha> va prod-latest.
• Healthcheck URL production chinh thuc.
• Rollback command/tag retention tren VM.

3.4 Kubernetes Backend - Optional/Reference Path

Kubernetes stack trong k8s/ hien duoc giu nhu optional path/reference, khong con la production path chinh sau thay doi ha tang sang VM:

• Base resources: StatefulSet, Service, Ingress, NetworkPolicy.
• Dev overlay: namespace aquax-dev, image tag dev-latest, host dev.api.aquax.vn, 1 replica, storage 5Gi.
• Prod overlay: namespace aquax-prod, image tag prod-latest, host api.aquax.vn, 2 replicas, storage 20Gi.
• Secret env duoc tao bang Kustomize secretGenerator tu .env.dev hoac .env.prod.
• Init container cho migration doi PostgreSQL san sang va chay npx prisma migrate deploy.
• Container chay non-root, read-only root filesystem, drop Linux capabilities.
• NetworkPolicy gioi han ingress tu NGINX/same namespace va egress toi DNS, PostgreSQL, Redis, HTTP/HTTPS external.

K8s chi nen dung khi du an quyet dinh quay lai GKE/Kubernetes hoac can moi truong song song.

3.5 Registry Va Server Deployment

Hien co hai luong CD backend trong repo, nhung production hien tai nen di theo VM/Docker Compose:

4. Moi Truong

5. Artifact Va Build Strategy

5.1 Backend API

Production artifact:

• Dockerfile: backend/Dockerfile.prod.
• Base image: Node.js 20 bookworm slim.
• Build stage: install dependencies, generate Prisma client, build NestJS.
• Runtime stage: copy dist, node_modules, modules, prisma, prisma.config.ts.
• Runtime command: node dist/main.js.
• Exposed port: 3001.

Manual build example:

cd backend
docker build --platform linux/amd64 -f Dockerfile.prod -t aquax-api .

5.2 Web Admin Portal

Current artifact:

• Dockerfile: web/Dockerfile.
• Command: npm run dev.
• Exposed port: 5173.

Production gap:

• Current Dockerfile is suitable for development/preview, not a finalized production static server.
• A production plan should choose one target: object storage/CDN, NGINX static container, managed frontend hosting, or Kubernetes deployment.
• Required build command already exists: npm run build, output should be served from Vite dist.

5.3 IoT Worker

Artifact:

• Dockerfile: iot-worker/Dockerfile.
• Multi-stage build with runtime command node dist/main.js.
• Uses IOT_WORKER_* variables for database, batching, auto-create sensor behavior, cloud forwarding and timeout.

Production/VM gap:

• Local Docker Compose path is confirmed.
• VM production broker-worker topology is TBD.
• Kubernetes topology is optional, not the current production assumption.

5.4 Mobile

Android:

• Expo app config: mobile/app.json.
• Package: com.aquax.app.
• Play Store upload script: mobile/scripts/deploy_android_playstore.sh.
• Script supports tracks: internal, alpha, beta, production.
• Script performs Expo prebuild, sets Android versionCode, builds AAB and uploads via Fastlane supply.

iOS:

• Bundle identifier exists: com.aquax.app.
• TestFlight/App Store release process is TBD.

5.5 AI Engine

Current state:

• ai-engine/README.md only identifies "AI Engine for AquaX".
• ai-engine/modules/train_isolation_forest.py and requirements.txt exist.

Deployment gap:

• No Dockerfile, API service, scheduled job, model registry, or runtime integration is confirmed for AI engine.
• AI deployment must remain planned until runtime architecture is implemented.

6. Configuration Va Secrets

6.1 Backend Environment

Backend .env.example confirms these groups:

• Database: DATABASE_URL.
• IoT ingest: IOT_INGEST_TOKEN, IOT_INGEST_AUTO_CREATE_SENSORS.
• MQTT device commands: MQTT_COMMAND_*.
• JWT/session: TTL, issuer, audience, JWT_KEYS_DIR.
• Redis: REDIS_HOST, REDIS_PORT, optional password.
• Email/password reset: reset URL, provider, Gmail/SMTP settings.
• Security/CORS/rate limits.
• Google Cloud Storage bucket and credentials path.

Production rules:

• Never commit real .env, .env.dev, .env.prod, service account JSON, keystore, Google Play JSON key, or SMTP credentials.
• IOT_INGEST_TOKEN, JWT settings, Redis password, DB password and email credentials must come from secret storage.
• K8s .env.dev and .env.prod are gitignored and generated from .env.example.
• POSTGRES_PASSWORD, DB_PASSWORD and DATABASE_URL must be kept consistent.

6.2 Web Environment

Web .env.example confirms:

• VITE_APP_NAME=AquaX.
• VITE_API_BASE_URL=http://localhost:3001/api.

Production rule:

• Web deployment must point VITE_API_BASE_URL to the correct API domain, for example https://api.aquax.vn/api for production.

6.3 VM Secrets

Production VM deployment must use an environment file or server-side secret mechanism outside git:

• .env must exist in the deploy directory on VM.
• DB, Redis, email, MQTT, IoT ingest and GCS credentials must be populated on the VM.
• Registry credentials should be provided through CI secrets or server credential store, not committed.
• Any service account JSON or mobile release secret must stay outside repository history.

6.4 K8s Secrets - Optional Path

K8s optional path uses:

• aquax-api-env generated from .env.dev or .env.prod.
• gcp-sa-key mounted at /app/gcp-credentials.
• GOOGLE_APPLICATION_CREDENTIALS=/app/gcp-credentials/service-account.json set in statefulSet patch.

Optional K8s gap/risk:

• JWT keys are mounted on emptyDir at /app/keys. This can regenerate keys after pod recreation and invalidate tokens. Production should use pre-provisioned key material via Secret, external key management, or another stable key strategy.

7. Database Migration Va Seed Strategy

7.1 Local

docker-compose.yml backend command currently runs:

npm ci &&
npx prisma generate &&
npx prisma migrate deploy &&
npx prisma db seed &&
npm run build &&
node dist/main

Implication:

• Local startup applies migrations and seeds automatically.
• This is useful for development.
• This should not be reused as-is for production because repeated seed execution can create drift or duplicate sample/demo data unless seed is idempotent and approved.

7.2 VM Production

Production VM deployment should run migrations as a controlled deployment step:

• Use npx prisma migrate deploy against production database before or during API rollout.
• Do not run npx prisma db seed in production unless the seed is explicitly approved and idempotent.
• Record migration output in deployment evidence.
• If migration fails, stop rollout and keep previous container running when possible.

7.3 Kubernetes - Optional

• Waits for PostgreSQL connection and credential validation.
• Runs npx prisma migrate deploy.
• Starts main API container only after migration succeeds.

Deployment rule:

• Every production release with schema changes must include reviewed Prisma migration files.
• Destructive migrations require explicit rollback/restore plan before production deployment.
• prisma db push is not allowed for production.

8. CI/CD Plan

8.1 Continuous Integration

Workflow: .github/workflows/ci.yml.

Triggers:

• Pull request to main or dev.
• Push to dev.

Checks:

Current CI gaps:

• Mobile lint/typecheck/build is not wired into root CI.
• IoT worker build/test is not wired into root CI.
• Container vulnerability scanning/signing is not confirmed.

8.2 VM CD

Current VM workflow: .github/workflows/cd-api-stunited-registry.yml.

Confirmed dev path:

1. Trigger on push to dev or manual dispatch.
2. Build backend/Dockerfile.prod.
3. Push dev-<date>-<sha> and dev-latest to docker-gateway.stunited.vn/aquax-api.
4. SSH to ST United VM.
5. Login to registry on VM.
6. In deploy dir, run docker compose -f docker-compose.yml pull aquax-api-dev.
7. Run docker compose -f docker-compose.yml up -d aquax-api-dev.
8. Run docker compose -f docker-compose.yml ps.

Production VM path to add/confirm:

• Branch/tag trigger for production, usually main, release tag, or manual approval.
• GitHub environment should be production.
• Image tags should be immutable, for example prod-<date>-<sha>, plus optional prod-latest.
• VM deploy dir and service name must be separate from dev.
• .env on production VM must be managed and backed up securely.
• Health check and rollback command must be part of workflow output.

8.3 GKE CD - Optional

Workflow: .github/workflows/cd-api.yml.

Optional GKE path:

1. Trigger on push to main for backend/k8s/workflow path changes.
2. Build backend/Dockerfile.prod.
3. Push image to GCP Artifact Registry.
4. Authenticate to GKE.
5. Apply Kustomize overlay.
6. Wait for statefulset/aquax-api rollout.
7. Rollback workflow job exists for production failure handling.

Known optional-workflow issue:

• The dev deploy job uses if: github.ref == 'refs/heads/dev', but the workflow currently triggers only on main. If GKE dev deployment is required later, the workflow trigger should include dev or the dev job should be moved to a separate workflow.

Required secrets:

• Registry user/password.
• SSH host/user/key.
• Optional deploy directory and compose file overrides.

9. Deployment Runbooks

9.1 Local Integrated Stack

Prerequisites:

• Docker and Docker Compose.
• Ports 3001, 5173, 5432, 6379, 1883 available.
• Env values reviewed for local use.

Run:

docker compose up --build

Verify:

curl http://localhost:3001/api/health
open http://localhost:5173

Stop:

docker compose down

9.2 Local IoT Stack

Run:

docker compose -f docker-compose.iot.yml up --build

Verify:

• PostgreSQL reachable on localhost:5433.
• MQTT reachable on localhost:1883.
• Telegraf and IoT worker logs show successful processing.
• Backend cloud forwarding works only when IOT_WORKER_CLOUD_INGEST_URL and token are configured.

9.3 VM Backend Deployment

Prerequisites:

• VM access by SSH or CI deploy key.
• Docker and Docker Compose installed on VM.
• VM can pull image from registry.
• Deploy directory contains docker-compose.yml and .env.
• Correct API service name confirmed, for example aquax-api-dev in the current dev workflow.
• Production should have backup and rollback tag confirmed before deployment.

Generic deploy flow:

cd /path/to/aquax/deploy
docker compose -f docker-compose.yml pull <api-service-name>
docker compose -f docker-compose.yml up -d <api-service-name>
docker compose -f docker-compose.yml ps
docker compose -f docker-compose.yml logs --tail=200 <api-service-name>

Verify:

curl https://<api-domain>/api/health

Rollback:

# Set previous image tag in VM compose/env, then:
docker compose -f docker-compose.yml pull <api-service-name>
docker compose -f docker-compose.yml up -d <api-service-name>

9.4 K8s Dev Backend - Optional

Prerequisites:

• GKE cluster access.
• kubectl context points to target cluster.
• Helm installed.
• NGINX Ingress and cert-manager installed.
• PostgreSQL and Redis Helm releases installed or upgraded.
• k8s/services-dev/.env.dev created from .env.example and secrets filled.
• gcp-sa-key secret created if Google Cloud Storage is used.

Deploy:

kubectl apply -k k8s/services-dev
kubectl -n aquax-dev rollout status statefulset/aquax-api

Verify:

kubectl -n aquax-dev get pods,svc,ingress
curl https://dev.api.aquax.vn/api/health

9.5 K8s Production Backend - Optional

Prerequisites:

• Production change approval.
• Release notes and migration notes reviewed.
• Backup/restore readiness confirmed for DB and uploaded files.
• k8s/services-prod/.env.prod prepared from .env.example.
• Domain api.aquax.vn and TLS issuance ready.
• Production secrets confirmed.

Deploy:

kubectl apply -k k8s/services-prod
kubectl -n aquax-prod rollout status statefulset/aquax-api

Verify:

kubectl -n aquax-prod get pods,svc,ingress
curl https://api.aquax.vn/api/health

9.6 Android Release

Prerequisites:

• Fastlane available or installable.
• mobile/secrets/build_config.env prepared.
• Android keystore present or env variables set.
• Google Play service account JSON available.
• Google Play service account has Release Manager access.

Run internal track:

cd mobile
./scripts/deploy_android_playstore.sh internal

Allowed tracks:

• internal
• alpha
• beta
• production

Release rule:

• Production track must only be used after QA sign-off, release notes, rollback/stop rollout plan and store metadata review.

10. Pre-Deployment Checklist

11. Post-Deployment Verification

Minimum smoke tests:

VM verification commands:

docker compose -f docker-compose.yml ps
docker compose -f docker-compose.yml logs --tail=200 <api-service-name>
curl https://<api-domain>/api/health

Optional K8s verification commands:

kubectl -n aquax-prod get pods
kubectl -n aquax-prod describe statefulset aquax-api
kubectl -n aquax-prod logs statefulset/aquax-api -c aquax-api --tail=200
kubectl -n aquax-prod logs statefulset/aquax-api -c db-migration --tail=200

12. Rollback Plan

Application rollback:

1. Identify last healthy image tag.
2. Confirm schema compatibility with current database.
3. For VM production, point Docker Compose/env to the previous healthy image tag.
4. Pull/restart the VM service with Docker Compose.
5. For optional K8s only, update image tag in Kustomize overlay and apply it.
6. Run smoke tests.
7. Record incident, impact and follow-up actions.

Database rollback:

• Prefer forward-fix migrations when possible.
• For destructive migration failure, restore from backup only after approval by Product, Engineering and Operations.
• Any irreversible migration must have a compensating plan before release.

VM rollback examples:

# On VM: point compose/env to the previous image tag, then restart service.
docker compose -f docker-compose.yml pull <api-service-name>
docker compose -f docker-compose.yml up -d <api-service-name>
docker compose -f docker-compose.yml logs --tail=200 <api-service-name>

Optional K8s rollback examples:

kubectl -n aquax-prod rollout history statefulset/aquax-api
kubectl -n aquax-prod rollout undo statefulset/aquax-api

Notes:

• VM rollback depends on immutable image tags and deploy directory state. Keep the last healthy tag in release evidence.
• StatefulSet rollback behavior must still be validated with the exact storage and migration state. Do not rely on image rollback alone when schema changed.

13. Monitoring, Logging Va Alerting

Signals to monitor:

• API /api/health.
• API latency and error rate.
• VM container health, restart count and Docker Compose service status.
• Pod restart count and rollout failures only when optional K8s path is used.
• PostgreSQL availability, slow queries, connection count and storage.
• Redis availability and memory.
• MQTT broker availability.
• Telegraf ingestion failures.
• IoT worker lag, batch failures and cloud forwarding failures.
• Email send failures.
• Report/file upload failures.
• Mobile crash/freezing metrics once provider is selected.

Current gap:

• No production monitoring provider, dashboard set, alert routing or SLO policy is confirmed in repo.

Recommended minimum alerts:

14. Disaster Recovery Va Backup

Recovery scope:

• PostgreSQL/TimescaleDB.
• Uploaded files/storage objects.
• Redis state if production behavior depends on persistent sessions.
• MQTT/IoT ingestion continuity.
• Backend API deployment artifacts and config.
• VM deploy directory, compose file and production .env recovery.

Current gap:

• RPO, RTO, backup frequency, retention period and restore drills are TBD.

Minimum policy to define before production:

15. Security Controls Trong Deployment

Confirmed controls:

• Backend production image uses backend/Dockerfile.prod.
• VM deploy process pulls image from registry and restarts service with Docker Compose.
• Secrets separated from code by .env.* and K8s Secret generation.
• CI enforces lint/build/test for backend and web.
• K8s-specific controls such as non-root container, read-only root filesystem, dropped capabilities, NetworkPolicy and cert-manager TLS remain available only on optional K8s path.

Controls to add/confirm:

• Container image vulnerability scanning.
• Image signing/provenance.
• Secret manager integration instead of local .env files for production.
• Stable JWT signing key management.
• Production backup encryption and access audit.
• Explicit deployment approval gates for production.

16. Known Gaps Va Action Items

17. Traceability